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INTRODUCTION  AND  SUMMARY  OF  PROJECT 


The  study  of  electronic  engineering  design  and  the 
influence  of  cognitive  style  upon  design  ability  has  been 
described  in  Reports  1 to  6 and  Scientific  Notes  1 and  2. 

In  summary,  designers  are  sampled  from  two  groups, 

A and  B;  namely,  experienced  designers  and  students  who  are 
skilled  in  electronics,  but  have  little  or  no  experience  of 
designing  on  their  own.  The  subjects  are  pre-tested  using  a 
lengthy,  computer  administered,  test  of  learning  style 
(Scientific  Note  No  1,  and  Progress  Report  No  6).  The  test 
is  scored  for  operation  learning  (ability  to  construct 
procedures  if  a goal  description  is  given),  comprehension 
learning  (ability  to  assimilate  a general  description,  or 
construct  it  from  other  general  aescriptions,  but  not 
necessarily  to  realise  the  entities  described)  and  versatility, 
which  is  an  index  of  ability  to  predict,  to  plan  and  to  realise 
plans.  Versatility  is  correlated  with  a combination  of  both 
operation  and  comprehension  learning  (though,  scorewise,  it 
need  not  be)  but  involves  additional  creative  or  productive 
skills.  Other  subscores  are  calculated  and  other  mental 
tests  are  administered,  together  with  a background  questionnaire, 
but  the  main  prediction  is  that  subjects  with  high  versatility 
scores  will  design  well. 

All  subjects  engage  in  a many  session  design  task 
which  involves  building  up  and  trying  out  a breadboard  prototype 
of  a sinulator  for  reaction  kinetics  and  chemical  equilibrium; 
used  for  classroom  demonstrations.  Any  design  must  satisfy  a 
brief  and  function  properly,  but  it  is  regarded  as  desirable  to 
maximise  reliability  and  transparency,  and  to  minimise 
environmental  sensitivity,  and  cost /weight /fragility.  Designs 
are  evaluated  according  to  these  and  other  criteria,  both  by 
independent  judges  and  the  subjects  themselves. 


I 


Of  the  two  groups,  nearly  all  experts  show  high 
versatility  scores.  Amongst  the  student  group,  there  is  greater 
variation  and  some  of  the  student  designers  with  lower 
versatility  are  given  (whilst  others  are  not  given)  specific 
versatility  training,  in  the  belief  that  it  will  improve  their 
performance.  s 

Behaviour  during  the  design  task  is  recorded 
conventionally  (by  sketches,  comments,  logging  the  concrete 
circuitry  built  up)  but  a more  penetrating  index  of  confidence 
is  obtained,  after  each  design  session  using  a computer  regulated 
interface,  known  as  THOUGHTSTICKER  (fully  described  in  Progress 
Report  No  4,  and  Progress  Report  No  6),  through  which  the  designer 
gives  a standard-format,  quantifiable  explanation  or  justification 
of  his  design  up  to  that  stage  in  the  task.  This  explanation  or 
justification  not  only  refers  to  the  electronic  circuitry,  itself 
(a  necessary  feature),  but  also,  very  often,  to  the  application 
domain;  for  example,  relevant  parts  of  theoretical  physical 
chemistry  and  the  problems  likely  to  be  encountered  in  demonstrat- 
ing concepts  or  practical  experiments  with  the  aid  of  the  simulator. 

The  format  of  the  explanation  and  justification  is 
graphical  and  called  an  entailment  mesh  (Scientific  Note  No  2, 
Progress  Report  No  6).  The  meshes  which  justify  the  completed 
design  are  finally  described  in  a systematic  manner  which  includes 
an  assignment  of  values  to  the  criterion  variables  (reliability, 
transparency,  etc). 

After  completing  the  design  task  as  individuals,  the 
subjects  are  assembled  into  representative  groups,  with  or  without 
expert  participants,  and  are  presented  with  a further  brief;  to 
devise  simulators  for  coisplex  reactions  in  which  the  prototypes 
they  have  all  (at  this  stage)  sianufactured  are  linked  into 
networks  and  certain  more  detailed  principles  underlying  reaction 
kinetics  are  deounstrated.  The  chief  objective  of  the  group 
sessions  is  to  sample  comsunication  between  designers,  their 
activity  as  a team,  and  the  influence  of  personal  style  and  design 
performance  upon  their  interaction.  Individuals  in  each  group 
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also  explain  or  justify  the  group  design  (again  using 
THOUGHTSTICKER)  as  seen  from  their  own  perspective.  Finally, 
because  standard-format  explanations  can  be  compared  and 
quantified,  the  group  task  forms  part  of  the  evaluation  of 
des ign-perf  ormance . 
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2.  Progress  Made 

Considerable  effort  was  expended  upon  finding  a 
design  task  that  would  be  considered  "realistic"  and,  having 
done  that,  in  removing  "blocks"  that  led  to  unduly  lengthy 
experimentation.  The  task,  as  finalised,  occupies  approximately 
10  sessions  and  about  40/50  hours  in  the  laboratory  for  expert 
designers,  and  6/8  sessions  or  a total  of  30/40  hours  for  student 
designers.  The  duration  of  the  team  task  is  variable;  the  group 
can  be  "fed"  problems  or  "prompted"  into  further  innovation  as 
long  as  their  behaviour  remains  coherent  and  reveals  interesting 
features. 


Up  to  date,  6 expert  designer . subjects  and  9 (non 
pilot)  student  subjects  have  completed  the  individual  task 
(reaction  simulator)  and,  at  the  moment  4 further  experts  and 
5 student  designers  are  starting  or  in  the  process  of  design. 

3 teams,  (maximum,  4 members)  are  meeting  periodically: 
though  full  attendance  is  not  required,  no  teams  have  flagged 
through  loss  of  motivation  and  it  is  intended  to  maintain  the 
sample  population  by  means  of  seminar-like  meetings,  for  the 
studies  described  in  our  proposal  EOARD-77-046  (which  has  recently 
received  technical  approval)  some  of  which  draw  subjects  from  the 
already  characterised  pool. 


3. 


Results  Obtained 


The  data  obtained  in  the  course  of  the  Individual 
design  task  is  exceptionally  detailed  (specimens  of  it  are  shown 
in  Scientific  Note  No  2).  The  session  by  session  data,  as  well 
as  the  final  "entailment  mesh"  (that  justifies,  or  explains,  a 
subject's  completed  design),  arc  open  to  numerical  quantification 
in  terms  of  complexity  and  connectivity  (specimen  calculations 
are  shown  as  part  of  Progress  Report  No  6)  and  the  descriptive 
evaluation  of  the  final  design,  together  with  its  justification, 
can  be  elicited  not  only  from  the  designer  responsible,  but  also 
from  other  subjects.  Hence,  various  detailed  and  numerical  cross 
comparisons  are  possible  and  noted,  also,  in  Progress  Report  No  6. 

The  difficulty  with  such  highly  informative  measures  is 
that  there  is  no  economic  way  of  using  them  until  a full  set  of 
data  is  available  from  the  individual  design  task  subjects.  At 
any  rate,  a proper  analysis,  to  form  the  reference  frame  in  which 
to  make  sense  of  team  task  data,  cannot  be  obtained  until  all 
individual  subjects  have  completed  their  assignment.  We  shall  be 
able  to  run  sample  analyses  about  six  weeks  from  this  date  (they 
will  be  presented,  as  soon  as  possible,  in  a Scientific  Note  No  3, 
containing  details  of  the  methods  and  computer  programs  employed). 
A full  analysis  of  the  individual  designers  should  appear  about 
one  month  after  the  next  Progress  Report  No  8. 

Using  far  less  refined  indices,  based  on  counts  of 
events,  evaluative  judgements  of  records,  etc,  some  results  can 
be  stated  at  this  juncture. 

The  subjects,  whether  experienced  designers  (Group  A) 
or  student  designers  (Group  B)  regard  the  brief  as  unusually 
comprehensive  and  detailed,  but  are  prepared  to  accept  it,  with 
this  caveat,  as  realistic.  The  expert  designers  recognise  the 
need  to  think  ahead  about  the  application  domain  from  the  outset 
and  typically  adopt  a global  perspective  during  the  1st  session; 


for  example,  most  of  them  start  out  by  defining  the  subproblems 
rates  and  computing  heat  exchange  and  temperature 
as  a function  of  the  extent  of  the  reaction  as  part  of  the  total 
design  problem.  Even  though  the  brief  encourages  such  a point  of 
view,  students  rarely  obtain  a global  perspective  until  the  3rd 
session.  Moreover,  they  are  not  infrequently  misled  into  confusion 
over  forward  and  back  reaction  rates. 


About  half  the  students  start  out  by  subscribing  to  a 
step-by-step,  "system  assembly"  point  of  view;  they  feel  they 
ought  to  arrive  at  a general  design  by  articulating  clear  cut 
parts,  such  as  "forward  rate  for  a component  reaction"  (a  couple 
of  students  have  paid  lip  service,  at  least,  to  a "flow  chart" 
design  technique).  With  one  possible  exception  the  experts  have 
been  aware  that  this  paradigm,  though  valuable,  is  not  directly 
applicable  for  a problem  of  this  complexity  and  refer  at  once  to 
pictures  of  the  application  domain  (frequently  developed  by  students 
at  a later  stage).  It  is  not  that  you  cannot  put  together  a general 
design  from  partial  designs;  some  people  do  so  successfully.  But 
in  doing  this  it  is  essential  to  choose  the  "right"  parts,  ie  parts 
that  are  representative  and  open  to  combination  (the  example  cited, 
forward  rate  of  a one  component  reaction,  is  not)  and  difficulties 
arise  because  inadequate  parts  are  identified.  In  general,  the 
identification  of  representative  parts  does  depend  upon  a global 
picture  (this  would  not  be  so  for  a trivial  task, in  which  the  right 
parts  are  "given".  ) 

Such  pictures  are  a crucial  ingredient  of  successful 
design  performance  but  may  also  be  misleading.  For  example, subjects 
fall  into  vicious  circularity  by  failing  to  recognise  that  many 
of  the  mathematical  expressions  in  the  brief  are  tautologous 
transformations  (they  are  all  of  them  valid)  and  represent  alterna- 
tive ways  of  saying  the  same  thing;  this  particular  misconception 
could  be,  and  ultimately  remedied  by  envisaging  reaction 

kinetics  and  equilibria  as  real  physical  chemistry,  rather  than 
the  text  book  mathematics  of  chemistry.  In  any  case,  this 
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particular  confusion  leads  subjects  to  compute  the  equilibrium 
constant  (K)  as  a function  of  Temperature  (T)  and,  at  the  same 
moment,  to  compute  Temperature  (T)  as  a more  or  less  sophisticated 
function  of  its  own  argument  in  the  original  design.  Experts  tend 
, to  avoid  or  minimise  this  kind  of  mistake;  presumably,  they  are 

more  alive  to  the  hazards  of  design.  However,  the  student  designers 
seem  to  learn  rapidly  from  making  mistakes  of  this  kind  and  to 
transfer  their  experience  to  avoid  other  possible  circularities  in 
the  design. 

Both  expert  designers  and  student  designers  have  trouble 
over  the  role  of  approximation.  It  is  shown  in  the  brief  that 
Temperature  (T)^  as  computed  in  the  required  simulator^  is  a "trick" 
variable.  All  that  is  demanded  is  a T function  that  satisfies 
certain  inequalities  and  changes  in  the  right  direction  except  f'"' 
a critical  region  where  more  accuracy  is  demanded.  This  is  an 
unavoidable  consequence  of  modelling  a complex  physical  chemical 
process  in  a simplified  manner  (in  the  team  task,  for  instance,  T 
computation  is  far  more  complex  and  the  "trick"  is  uncovered  by 
deeper  examination  of  the  processes).  Subjects  eventually  come  to 
terms  with  the  limits  of  a simulation  but  do  so  in  different  ways 
j with  more  or  less  direct  reference  to  the  application  domain. 

Here,  the  use  of  valid  analogical  reasoning  (Section  4)  appears 
to  be  especially  important. 

I 

Some  general  findings  are  as  follows  : 


1).  There  is  a significant  positive  correlation  between 
versatility  scores  on  the  test  of  learning  style  and  the  extent  to 
which  subjects  take  account  of  the  application  domain;  this 
probably  correlates  positively  with  any  plausible  index  of  design 
quality. 


2).  Students  submit  completed  designs  sooner  than  experts: 
they  also  make  more  self  corrected  mistakes  (self  correction 
because  a circuit  arrangement  is  found  not  to  work  satisfactorily) 
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than  experts,  and,  consequently,  backtrack  more  often.  Back- 
tracking takes  up  time  but  either  inexperienced  (student)  designers 
learn  rapidly  from  mistakes,  or  experienced  designers  are  overly 
self  critical,  for  the  average  total  design  time  occupied  by 
students  is,  as  stated,  less  than  the  average  for  expert  designers. 

3).  In  interpreting  these  results,  it  is  essential  to 
recall  chat  every  design  accepted  as  being  completed  must  satisfy 
the  brief  and  consequently  various  performance  tests  that  are 
built  into  the  brief.  Looked  at  grossly,  there  are  only  a few 
final  design  types,  though  the  details  differ.  The  main  and 
outstanding  contrast  is  between  the  process  of  designing  and  the 
justifications  or  explanations  offered  by  the  subjects  at  various 
stages  in  the  design  task  including,  of  course,  their  justification 
or  explanation  of  the  (acceptable)  <^inalised  design.  There  is  no 
doubt  that  experts^  not  surprisingly,  achieve  greater  elegance,  but 
they  take  a longer  time  to  do  so  and  the  effort  may  or  may  not  (in 
this  case  we  must  rely  upon  the  full  analysis)^ be  worthwhile. 


4).  Apart  from  the  backtracking  behaviour  noted  in  (2) 
above,  the  crudely-observed  consistent  differences  in  style  between 
student  designers  and  experts,  do  not  appear  to  be  greater  than  the 
consistent  differences  between  students  or  between  experts  (the 
within-group  differences).  These  are  reflected  in  and  predictable 
from  all  the  scores  on  the  initial  test  for  learning  style 
(operation  learning,  comprehension  learning  and  versatility). 


5).  Subjects  with  a high  versatility  score  (and  often 
with  a high  comprehension  score)  grasp  certain  features  of  the 
circuit  itself  and  its  relation  to  the  application  domain  more 
rapidly  than  others.  Amongst  the  student  designers^ this  tendency 
shows  up  as  more  rapid  backtracking  (remedying  mistakes,  possibly; 
though,  here,  we  must  rely  upon  a detailed  analysis,  in  recognising 
mistakes).  It  also  shows  up  in  terms  of  overcoming  misconceptions 
about  part  of  the  application  domain, (the  interpretation  of 
equations  in  terms  of  physical  chemistry  theory). 


4.  Analytical  Reasoning,  Insight  and  Innovation 


One  definite  outcome  of  the  research,  even  before 
detailed  analysis  is  carried  out,  is  that  analogical  reasoning 
is  ubiquitous.  The  use  and  occasional  misuse  of  analogy  is 
frequent  and  in  contrast  to  the  exponents  of  an  algorithmic 
model  of  design,  a prominent  part  of  both  student  and  expert 
behaviour.  Analogies  may  be  recognised  or  abduced  (invented) 
either  within  the  circuitry  which  is  designed  or  between 
components  of  the  design  and  features  or  relations  in  the 
application  domain.  Insofar  as  there  are  many  different  kinds 
of  analogy,  for  example,  quite  distinct  analogies  of  form  (or 
result)  and  analogies  of  method,  this  finding  is  not  altogether 
at  odds  with  those  observers  who  emphasise  the  algorithmic/ 
heuristic/systematic  aspect  of  design  (for  instance,  one  kind  of 
analogy  of  method  is  to  recognise  that  an  algorithm  or  operation 
used  in  one  field,  may  be  fruitfully  employed  in  another,  some- 
times to  achieve  a similar  result,  when  there  is  also  an  analogy 
of  form,  sometimes  to  yield  a different,  but  still  useful,  result). 

The  finding  is  obtrusive.  Probably  the  merit  of 
versatility  (noted  in  Section  3,  1)  is  chiefly  due  to  the 
successful  use  of  analogical  reasoning  by  versatile  subjects,  in 
contrast  to  the  misuse  of  analogy  (abduction  of  similitudes  that 
are  trivial  or  are  not  valid)  by  the  less  versatile  students. 

In  order  to  have  confidence  in  such  a finding,  it  is 
necessary  to  underpin  it  with  a proper  theory  of  analogies, 
their  types,  the  difference  between  recognising  and  inventing 
them,  etc.  Such  a theory  exists,  though  it  has  not  been  exploited 
or  discussed  in  previous  progress  reports.  On  my  recent  visit  to 
Wright  Patterson  Air  Force  Base,  in  Dayton,  Dr  G Klein  requested 
a technical  account  of  the  underpinning  theory  as  part  of  the  report 
sequence.  Since  the  material  may  be  of  more  general  interest,  the 
technical  account  of  analogy  is  to  be  issued  as  a Scientific  Note 


No  3 shortly  after  this  ]^r ogress  Report  is  submitted 


Versatility  Training 


Until  a full  analysis  is  available,  the  question  of 
whether  or  not  versatility  training  helps  designers  will  remain 
open.  On  the  face  of  it, we  are  unlikely  to  obtain  a positive 
result  for  two  reasons.  One,  which  was  anticipated  as  a 
possibility,  is  that  few  students  competent  to  undertake  the 
design  task  have  a very  low  versatility  score,  so  that 
differential  sampling  is  a real  problem.  The  other  reason,  just 
as  strong, but  not  fully  anticipated,  is  that  the  experimental 
method  developed  to  exteriorise  the  design  process,  acts  as  a 
£orm  of  versatility  training  in  its  own  right.  Both  students 
and  experts  are  caused  (in  the  analytic  sessions,  when  designs 
are  explained,  or  justified),  to  contemplate  their  own  mental 
mechanisms  and  areas  of  strength  and  weakness.  To  do  so 
systematically  is  part  of  versatility  training  so  that,  under 
these  circumstances,  the  specific  training  would  have  to  produce 
a very  high  magnitude  effect  in  order  to  provide  a significant 
comparative  result.  This  is  not  evidence  that  versatility 
training,  as  such,  is  ineffective:  rather  that  the  lengthy 
monitoring  procedure  is  a mild  form  of  versatility  training  and 
that  relatively  short  periods  of  intense  training  are  likely  to 
add  relatively  little  to  the  overall  performance. 


II 
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6.  Implications  for  Design  Training 

Two,  commonly  stressed,  philosophies  of  design  are 
noted  in  Section  4;  one  emphasising  the  systematic  application  of 
principles  or  operations  or  algorithms  either  for  testing  or 
transforming  a partly  completed  design;  the  other  emphasising  the 
role  of  insight,  innovation  and  some  kinds  of  analogical  reasoning. 
There  is  also  a prevailing  impression  (rather  than  an  assertion) 
that  algorithmic  skills  and  operations  can  be  inculcated  or  improved 
by  instruction,  whereas  little  can  be  done  to  improve  insight  or 
analogical  inventiveness.  In  view  of  this,  the  finding  of  Section  4 
(that,  when  design  is  studied  in  depth,  analogical  reasoning  appears 
as  a dominant  mode)  might  be  misconstrued  as  a doctrine  of  despair. 


It  is  wise  to  state,  categorically,  at  this  juncture, 
that  we  hold  quite  a different  view.  On  the  one  hand,  there  does 
not  seem  to  be  a rigid  demarcation  between  the  design  philosophies. 
Considered  in  sufficient  detail  (for  example,  by  investigations  that 
are  designed  to  exteriorise  normally  unobservable  aspects  of 
cognition,  hypotheses  formation,  and  the  like;  ie.  studies  such 
as  the  present  study),  the  heuristic/algorithmic  and  the  innovative/ 
analogical  orientations  are  complementary  rather  than  opposed. 
Surely,  there  are  consistently  different  design  styles  but  there  is 
an  algorithmic/heuristic  and  an  analogical/innovative  aspect  to 
each  competent  style. 


Next,  we  definitely  deny  that  analogical  reasoning  and 
probably  innovative  reasoning  (which  does  appear  to  be  both  commonly 
used  and  effective)  is  somehow  unlearnable.  Design  training  may 
have  to  be  augmented  to  take  account  of  this  type  of  thinking,  but 
it  is  no  more  nor  less  difficult  to  train  people  in  the  use  of 
analogy  (conversely,  in  avoiding  the  misuse  of  analogy)  than  it  is 
to  teach  them  helpful  algorithms  and  principles.  The  belief,  if 
it  exists,  that  insight  and  analogy  are  ineluctable,  is  frankly 
mistaken  and  due  to  the  lack,  hitherto,  of  an  adequate  theoretical 
basis  for  analogical  reasoning. 
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Either  the  theoretical  basis  outlined  in  Scientific 
Note  No  3 or  some  equally  precise  statement,  lead  directly  to 


appropriate  training  expedients;  possibly  identical  with 
"versatility  training"  or  "Learning  to  Learn".  It  is  true  that 
positive  recomnendations  are  seldom  encountered  (presumably 
because  of  uncertainty  about  the  mental  mechanisms  involved  and 
the  absence  of  a theory  about  what  should  be  achieved).  On  the 
other  hand,  there  is  ample  evidence  that  cogent  training 
procedures  can  be  instituted  and  that,  when  instituted,  they  are 
effective;  the  evidence  in  question  comes  from  depth  studies 
bringing  individual  differences  into  the  picture,  either  those  of 
other  laboratories  or  our  own. 


Plans  for  Continuation  of  the  Work 


Individual  and  team  studies  are  continuing  and  an 
adequate  team  task  period  is  secured  by  item  (a)  in  the  work 
statement  of  the  continuation  proposal  EOARD-77-046. 

Scientific  Note  No  4 is  due  for  almost  immediate  delivery  and  is 
partly  typed  at  the  moment.  Progress  Report  No  8 will  contain  or 
refer  to  a detailed  analysis,  though  greater  refinement  is  possible 
under  item  (e)  (evaluation  study).  In  the  meanwhile,  sample 
analysis  data  will  be  furnished,  together  with  a statement  of  the 
methods  used  in  Scientific  Note  No  3. 

As  soon  as  possible  ,we  shall  start  work  statement  (c) 
(number  of  alternative  designs,  different  but  meaningful  tasks) 
in  order  to  retain  members  of  the  subject  pool.  For  item  (d) 
(comparison  of  system  programming  techniques),  it  is  desirable  to 
have  at  least  some  of  the  present  subjects  available,  which  bears 
directly  upon  the  theme  of  Section  5 in  the  report.  In  contrast, 
item  (b)  (adaptive  stylistic  tests)  and  the  finer  points  of 
item  (e)  have  less  urgency  in  this  respect,  though  they  are  of 
comparable  significance. 


